Online machine learning for calibration of autonomous earth moving vehicles

ABSTRACT

In some implementations, the EMV uses a calibration to inform autonomous control over the EMV. To calibrate an EMV, the system first selects a calibration action comprising a control signal for actuating a control surface of the EMV. Then, using a calibration model comprising a machine learning model trained based on one or more previous calibration actions taken by the EMV, the system predicts a response of the control surface to the control signal of the calibration action. After the EMV executes the control signal to perform the calibration action, the EMV system monitors the actual response of the control signal and uses that to update the calibration model based on a comparison between the predicted and monitored states of the control surface.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. Application No. 17/748,999,filed May 19, 2022, which is a continuation of U.S. Application No.17/359,432, filed Jun. 25, 2021, now Patent No. 11,352,769, which isincorporated by reference in its entirety.

BACKGROUND Field of Art

The following disclosure relates generally to a method for movingmaterial to, from, or within a work site using autonomous heavymachines.

Description of the Related Art

Heavy machines and/or construction vehicles, for example backhoes,loaders, and excavators, (generally referred to herein as “earth movingvehicles”) are often used to move and manipulate large quantities ofearth or other similar material. Currently, operation of earth movingvehicles is very expensive as each vehicle requires a manual operator beavailable and present during the entire process to control the earthmoving vehicle. Further, implementing autonomous control of an earthmoving vehicle presents other challenges because of changeableconditions on many work sites and the complexity and variety of earthmoving vehicle models.

Many work sites are outdoors, where soil conditions can dramaticallychange day to day and hour to hour due to the weather, throughexcavation to lower layers of material, or the like. The properties ofsoil being moved or excavated can alter the optimal strategy for anearth moving vehicle interacting with the soil. Similarly, many earthmoving vehicles need to coordinate multiple control surfaces to performeven simple-sounding tasks. For example, excavators generally havemulti-element arms with multiple joints that are coordinated to use theexcavator bucket. Further, the same action may be able to be performedin a variety of ways depending on the current conditions.

Additionally, there is a large variety of specialized earth movingvehicle models, each with different control surface layouts, actuationstrengths, and other characteristics which make it difficult to developan autonomous control system that can effectively control multiple typesof earth moving vehicle. Similarly, many earth moving vehicles use amixture of actuation and control methods (such as, hydraulic, electronicactuation, pneumatic, and the like) which each may react differently tocontrol input.

SUMMARY

Described herein is an autonomous or semi-autonomous earth moving systemincluding a control system, sensors, and an earth moving vehicle (EMV)for excavating or performing other tasks on a work site. The earthmoving system controls and navigates the earth moving vehicle within awork site to autonomously perform earth moving routines or otheractions. The earth moving system uses a combination of sensorsintegrated into the earth moving vehicle to record the positions andorientations of the various components of the earth moving vehicleand/or the conditions of the surrounding environment. Data recorded bythe sensors may be aggregated or processed in various ways, for example,to generate digital representations of a work site and instructions forexcavating earth from the work site, determine and control the movementof the earth moving vehicle over routes within the work site, andperform other tasks described herein.

An autonomous earth moving system can select an action for an earthmoving vehicle to autonomously perform using a tool (such as anexcavator bucket). The system then generates a set of candidate toolpaths, each illustrating a potential path for the tool to trace as theearth moving vehicle performs the action. In some cases, the system usesan online learning model iteratively trained to determine whichcandidate tool path best satisfies one or more metrics measuring thesuccess of the action. The earth moving vehicle then executes the earthmoving action using the selected tool path and measures the results ofthe action. In some implementations, the autonomous earth moving systemupdates the machine learning model based on the result of the executedaction.

Similarly, the autonomous earth moving system can determine a desiredstate for a portion of the EMV including at least one control surface.Then the EMV selects a set of control signals for moving the portion ofthe EMV from the current state to the desired state using a machinelearning model trained to generate control signals for moving theportion of the EMV to the desired state based on the current state.After the EMV executes the selected set of control signals, the systemmeasures an updated state of the portion of the EMV. In some cases, thisupdated state of the EMV is used to iteratively update the machinelearning model using an online learning process.

When an EMV performs an action comprising moving a tool of the EMVthrough soil or other material, the EMV can measure a current speed ofthe tool through the material and a current kinematic pressure exertedon the tool by the material. Using the measured current speed andkinematic pressure, the EMV system can use a machine learned model todetermine one or more soil parameters of the material. The EMV can thenmake decisions based on the soil parameters, such as by selecting a toolspeed for the EMV based on the determined soil parameters.

In some implementations, the EMV uses a calibration to inform autonomouscontrol over the EMV. To calibrate an EMV, the system first selects acalibration action comprising a control signal for actuating a controlsurface of the EMV. Then, using a calibration model comprising a machinelearning model trained based on one or more previous calibration actionstaken by the EMV, the system predicts a response of the control surfaceto the control signal of the calibration action. After the EMV executesthe control signal to perform the calibration action, the EMV systemmonitors the actual response of the control signal and uses that toupdate the calibration model based on a comparison between the predictedand monitored states of the control surface.

The features and advantages described in the specification are not allinclusive and, in particular, many additional features and advantageswill be apparent to one of ordinary skill in the art in view of thedrawings, specification, and claims. Moreover, it should be noted thatthe language used in the specification has been principally selected forreadability and instructional purposes and may not have been selected todelineate or circumscribe the inventive subject matter.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is an environment diagram showing an autonomous earth movingsystem, according to one embodiment.

FIG. 2A illustrates an example placement of sensors for a compact trackloader, according to one embodiment.

FIG. 2B illustrates an example placement of sensors for an excavator,according to one embodiment.

FIG. 3 is a block diagram of an autonomous control system of an earthmoving vehicle, according to one embodiment.

FIG. 4 illustrates an example subdivision of a routine into operationsand control signals for an earth moving vehicle to perform, according toone embodiment.

FIG. 5 is a block diagram of an example machine learning soil parametermodel for an earth moving vehicle, according to one embodiment.

FIG. 6 illustrates example placement of control surfaces for anexcavator, according to one embodiment.

FIG. 7 illustrates example tool path parameters of a scooping operationfor an excavator, according to one embodiment.

FIG. 8 is a block diagram of an example machine learning tool path modelfor an earth moving vehicle, according to one embodiment.

FIG. 9 is a block diagram of an example machine learning control modelfor an earth moving vehicle, according to one embodiment.

FIG. 10 is a block diagram of an example machine learning calibrationmodel for an earth moving vehicle, according to one embodiment.

FIG. 11 is a flowchart describing a process for determining soilparameters of soil interacted with by an earth moving vehicle using amachine learning soil parameter model, according to one embodiment.

FIG. 12 is a flowchart describing a process for selecting a tool pathfor an earth moving vehicle using a machine learning tool path model,according to one embodiment.

FIG. 13 is a flowchart describing a process for controlling an earthmoving vehicle using a machine learning control model, according to oneembodiment.

FIG. 14 is a flowchart describing a process for calibrating an earthmoving vehicle using a machine learning calibration model, according toone embodiment.

FIG. 15 is a high-level block diagram illustrating an example of acomputing device, according to one embodiment.

The figures depict various embodiments of the present invention forpurposes of illustration only. One skilled in the art will readilyrecognize from the following discussion that alternative embodiments ofthe structures and methods illustrated herein may be employed withoutdeparting from the principles of the invention described herein.

DETAILED DESCRIPTION Autonomous Earth Moving System

An earth moving system, as used herein, is a system including one ormore earth moving vehicles (EMVs) capable of autonomously orsemi-autonomously manipulating earth, debris, or other material at awork site. Herein, the term “material” refers to earth (dirt, soil,gravel, construction aggregate, and the like) and/or other materials(for example, boards, concrete, other construction supplies, waste, ordebris) which an EMV (or other aspect of an earth moving system) mayinteract with using a tool of the EMV. Though some implementations aredescribed herein in relation to earth or soil, this description is notintended to exclude implementations involving the manipulation of othermaterials, such as construction aggregate and/or building supplies. Forexample, the terms “earth moving” and “moving earth” may apply both tothe movement of dirt, soil, or gravel as well as to the movement ofnon-earth materials, such as boards or concrete. For simplicity, theterms “earth moving,” “moving earth,” “earth moving action,” and thelike as used throughout the following description are not intended to belimiting to any one context or type of material.

FIG. 1 is an environment diagram showing an autonomous earth movingsystem, according to one embodiment. The environment 100 of FIG. 1includes an autonomous EMV system 110, a network 170, and an off-unitcontrol system 180 which can communicate with the autonomous EMV system110. The autonomous EMV system 110 includes an autonomous control system120 which can interface with an earth moving vehicle (EMV) 130 and asuite of sensors 160. The sensors 160 can include sensors mounted onand/or around the EMV 130 (or otherwise providing information on thework site). In some implementations, the sensors 160 include sensorsmounted on the EMV 130 to record data describing the state of the EMVand the work site. For example, the sensors 160 of environment 100includes measurement sensors 162, spatial sensors 164, imaging sensors166, and position sensors 168 for monitoring the state of the EMV 130.As examples, FIGS. 2A and 2B (which will be discussed further below)illustrate the example placement of sensors for, respectively, a compacttrack loader and an excavator.

Each of these components will be discussed further below in relation toFIGS. 1-3 . Although FIG. 1 illustrates only a single instance of mostof the components of the autonomous EMV system 110, in practice morethan one of each component may be present, and the EMV system 110 orenvironment 100 may include additional, fewer, or different componentsthan those described herein.

Earth Moving Vehicle

As used herein, an earth moving vehicle (EMV) such as the EMV 130 is avehicle designed to manipulate construction material on a work site.EMVs are typically large and capable of moving and/or manipulating largevolumes of construction material at a single time, particularly relativeto what an individual human can move by hand.

As used herein, “earth moving” refers generally to moving or shapingearth or other material within a work site, for example digging orfilling a hole, leveling a mound, excavating, or depositing a volume ofearth or materials from a first location to a second location. In someimplementations, earth moving vehicles 130 excavate earth by scraping ordigging earth from beneath the ground surface. Examples of earth movingvehicles 130 within the scope of this description include, but are notlimited to, loaders such as backhoe loaders, track loaders, wheelloaders, skid steer loaders, scrapers, graders, bulldozers, compactors,excavators, mini-excavators, trenchers, skip loaders, and other types ofconstruction machines.

Among other components, earth moving vehicles 130 generally include achassis or frame upon on which all other components are physicallymounted, a drive system which gives the EMV 130 mobility through thework site, and one or more engines or other power sources. Additionally,an EMV 130 may include a control input 140 allowing control over theoperation of the EMV 13, a set of control surfaces 150 (including an EMVtool 155) controlled by the control input 140, and one or more sensors160 mounted on or around the EMV 130.

Each control surface 150 represents one or more parts or portions of theEMV 130 that can be moved or otherwise controlled based on controlsignals received by the control input 130 of the EMV 130. A controlsurface 150 can be manipulated through the control input 140 to maneuverthe EMV 130, perform an earth moving action, or otherwise change thestate of the EMV 130. The number and function of each control surfaces150 depend on the type and purpose of the EMV 130. For example, askid-steer vehicle may include separate control surfaces 150 for theleft and right tracks of the EMV, whereas a wheeled EMV may includethrottle and steering control surfaces. Example control surfaces 150include but are not limited to: throttles and propulsion controls,brakes, articulated joints (such as hydraulicly actuated joints), andelectronic devices (such as lights or sensors 160).

Similarly, the EMV 130 may include control surfaces 150 associated witha tool 155 used to manipulate material and/or perform earth movingactions. In some implementations, different aspects of the tool 155 arecontrolled by different control surfaces 150, including not only controlsurfaces for the instrument for manipulating material (such as excavatorbucket for collecting dirt), but also one or more articulated elementsfor positioning and operating the instrument. For example, in anexcavator or loader, the tool 155 can be associated with controlsurfaces 150 for the bucket itself (for example, controlling therelative angle of the bucket or a “thumb” of the bucket) in addition toseparate control surfaces 150 for the joints of the multi-element armthat adjusts the position and orientation of the bucket.

In some embodiments, the EMV 130 accepts instructions for controllingone or more control surfaces 150 through the control input 140. Thecontrol input 140 may be configured for manual control by a humanoperator or can be linked to the autonomous control system 120 forautonomous control. The control input 140 of an EMV can include multiplecontrols corresponding to different control surfaces 150 using a varietyof methods to receive control signals. For example, control signals canbe received electronically (as in controls for electronics such aslights or in “drive-by-wire” systems) or via mechanical input (such asthrough motion of a mechanical linkage or by applying force to ahydraulic system). A control input 140 can include a set ofjoysticks/levers controlling the EMV’s drive system, tool 155, and anyother directly controllable control surface 150.

In some implementations, the engine powers both the drive system and theoperation of the control surfaces 150. The engine may be an internalcombustion engine, or an alternative power plant, such as an electricmotor or battery. In many earth moving vehicles 130, the engine powersthe drive system and a hydraulic system used to actuate the tool 155. Acommon property of hydraulic systems used within earth moving vehicles130 is that the hydraulic capacity of the earth moving vehicle 130 isshared between the drive system and other control surfaces 150. In someembodiments, the instructions and control logic for the earth movingvehicle 130 to operate autonomously and semi-autonomously includesinstructions relating to determinations about how and under whatcircumstances to allocate the hydraulic capacity of the hydraulicsystem.

Sensor System

As introduced above, the set of sensors 160 includes a combination ofmeasurement sensors 162, spatial sensors 164, imaging sensors 166,position sensors 168, and/or other types of sensors. In someimplementations, sensors 160 are configured to collect data related toan earth moving vehicle 130, environmental data surrounding the earthmoving vehicle 130, or environmental data about the work site. In someimplementations, sensors 160 are either removably mounted to the EMV 130or integrated into the EMV (either as a modification or as factoryequipment from the manufacturer of the EMV 130).

Sensors 160 can transmit collected data, in real-time or as soon as anetwork connection is achieved, automatically without input from theearth moving vehicle 130 or a human operator. Sensor 160 may beconfigured to communicate received data to the autonomous control system120 and/or the off-unit control system 180. In some implementations,sensor data is transmitted to both control system 120 and 180, such thatrecorded data arrives at the off-unit control system 180 for consumptionand decision making by a manual operator while also providing the datato the autonomous control system 120 for autonomous or semi-autonomousdecision making and control of the EMV 130.

Data collected by sensors 160 can be used by the autonomous controlsystem 120 and/or the off-unit control system 180 to inform theoperation of the EMV 130 (for example, for determining and performingone or more earth moving routines and/or actions). For example, theautonomous control system 120 can control the drive system of the EMV130 to reposition the EMV based on environmental data and/or a locationof the earth moving vehicle 130 gathered by a sensor 160.

An individual sensor 160 or group of sensors 160 may include its ownnetwork adapter (not shown) that communicates with the control systems120 and 180 through either a wired or wireless connection to send sensordata, change modes, or the like. For wireless connections, the networkadapter may be a Bluetooth Low Energy (BTLE) wireless transmitter,infrared, or 802.11 based connection. For example, if the networkadapter of the sensor assembly 110 is configured to communicate via awireless standard such as 802.11 or LTE, the network adapter mayexchange data with a wireless access point such as a wireless router,which may in turn communicate with the control system 120 and 180. Forwired connection, a wide variety of communications standards and relatedarchitecture may be used, including Ethernet, a Controller Area Network(CAN) bus, or similar.

Depending on the implementation, individual sensors 160 and the sensorsuite as a whole range in complexity from simplistic measurement devicesthat output analog signals electrically coupled to a network bus orother communicative network to more complicated devices which includetheir own onboard computer processors, memory, and communicationsadapters to gather, process, and transmit sensor data. Regardless ofconstruction, the sensors 160 together function to record, store, andreport information to the autonomous control system 120 and/or off-unitcontrol system. In some implementations, a timestamp is associated withrecorded sensor data, either by the recording sensor 160 at the time ofcollections, or by the autonomous control system 120/off-unit controlsystem 180 as the sensor data is received.

Measurement sensors 162 generally measure properties of the ambientenvironment, or properties of the earth moving vehicle 130 itself. Theseproperties may include tool and control surface position/orientationand/or relative articulation, vehicle speed, ambient temperature,hydraulic pressure, and the like. A variety of possible measurementsensors 168 may be used, including hydraulic pressure sensors, linearencoders, radial encoders, inertial measurement unit sensors, inclinesensors, accelerometers, strain gauges, gyroscopes, and string encoders.

Spatial sensors 164 may collect information about the relative positionof object in the environment around the EMV 130. For example, a spatialsensor 164 may output a three-dimensional map in the form of athree-dimensional point cloud representing distances between one meterand fifty meters between the spatial sensors 164 and the ground surfaceor any objects within the field of view of each spatial sensor 164. Inone embodiment, spatial sensors 164 include a set of light emitters(e.g., Infrared (IR)) configured to project structured light into afield near the earth moving vehicle 130, a set of detectors (e.g., IRcameras), and a processor configured to transform data received by theinfrared detectors into a point cloud representation of thethree-dimensional volume captured by the infrared detectors as measuredby structured light reflected by the environment. In one embodiment, thespatial sensor 164 is a LIDAR sensor having a scan cycle that sweepsthrough an angular range capturing some or all of the volume of spacesurrounding the earth moving vehicle 164. Other types of spatial sensors164 may also be used, including time-of-flight sensors, ultrasonicsensors, and radar sensors.

Imaging sensors 166 capture still or moving-video representations of theground surface, objects, and environment surrounding the earth movingvehicle 130. Examples of imaging sensors 166 include, but are notlimited to, stereo RGB cameras, structure from motion cameras, andmonocular RGB cameras. In one embodiment, each image sensor 166 canoutput a video feed containing a sequence of digital photographic imagesat a rate of 20 Hz. In one embodiment, multiple imaging sensors 166 aremounted such that each imaging sensor 166 captures some portion of theentire 360-degree angular range around the vehicle. For example, front,rear, left lateral, and right lateral imaging sensors 166 may be mountedto capture the entire angular range around the earth moving vehicle 130.

Position sensors 168 provide a position of the earth moving vehicle 130.This may be a localized position within a work site, or a globalposition with respect latitude/longitude, or some other externalreference system. In one embodiment, a position sensor is a globalpositioning system interfacing with a static local ground-based GPS nodemounted to the earth moving vehicle 130 to output a position of theearth moving vehicle 130.

There are numerous different ways for the sensor assembly 160 generallyand the individual sensors specifically to be constructed and/or mountedto the earth moving vehicle 130. This will also depend in part on theconstruction and type of the earth moving vehicle 130. Using the compacttrack loader of FIG. 2A as an example, the representations with diagonalcrosshatching represent the example placements of a set of measurementsensors 162, the representation with diamond crosshatching representexample placements of a set of spatial sensors 164, and therepresentations with grid crosshatching represent example placements ofa set of position sensors 168. Using the excavator shown in FIG. 2B asanother example, diagonal crosshatchings represent measurement sensors162 (for example, the vertical crosshatchings near the drive system 210represent example placements for a linear encoder 210 measuring therelative rotation of the tracks and chassis), diamond crosshatchingsrepresent spatial sensors 164, grid crosshatchings represent positionsensors 168, and horizontal crosshatchings near the roof representimaging sensors 166 (for example RGB cameras).

Autonomous Control System

An autonomous control system 120 can include computer hardware (forexample, microcontrollers, computers, and/or computing devices) capableof interpreting sensor data and/or commands from an off-unit controller180 and interfacing with the EMV 130 to perform actions. As describedabove, data collected by sensors 160 is communicated to the autonomouscontrol system 120 to assist in the control of the EMV 130. In someimplementations, an autonomous control system 120 includes interfacehardware allowing the ACS 120 to send various types of control signalsto the control input 140 of the EMV 103. Interfaces between theautonomous control system 120 and the EMV 130 can be electronic (to sendanalog and/or digital control signals), mechanical (using servo motorsor the like to physically manipulate one or more controls), or any othersuitable interface method.

In some embodiments, the autonomous control system 120 is implemented asa modification or retrofit to the EMV 130 giving the EMV 130 thecapability for autonomous or semi-autonomous control. In otherimplementations, the EMV 130 is designed and/or manufactured with anintegrated autonomous control system 120. In some implementations, theaddition of the ACS 120 includes the replacement of some aspects of thecontrol input 140 or the addition of new controls to the control input140 for interfacing with the autonomous control system 120.

In some implementations, the autonomous control system 120 controls theEMV 130 through the selection and execution of one or more earth movingroutines. An “earth moving routine,” as used herein, is a set ofcomputer instructions that coordinates control of the various controlsurfaces 150 of the earth moving vehicle 130 to accomplish a task on thework site. For example, earth moving routines can accomplish work sitepreparation, digging or excavation objectives (including digging,dumping, grading, filling, trenching, compacting, backfill, wall cutbacketc.), fill estimation and volume checks for a tool of the EMV 130,and/or EMV calibration objectives.

In some implementations, earth moving routines are subdivided into“earth moving actions” representing individual actions that areperformed (sometimes repetitively) by an EMV 130 during the earth movingroutine. For example, an earth moving routine for excavating a specificarea (such as trenching from point A to point B) can be associated withearth moving actions for scooping dirt from a target location,maneuvering to an appropriate dump location, and dumping the excavateddirt, among others. In some implementations, the autonomous controlsystem 120 dynamically selects earth moving actions (and parameters forthose earth moving actions) to accomplish a target earth moving routinebased on the current state of the work site and EMV 130. For example,earth moving actions can include scooping or dumping material with atool 155, performing a maneuver (such as a turning the EMV 130), orperforming tool-specific actions (such as jerking an excavator bucket tolevel the material in the bucket).

The autonomous control system 120 can be designed to control an EMV 130to carry out an earth moving routine either entirely autonomously orsemi-autonomously. In some implementations, a semi-autonomouslyoperating EMV 130 not only responds to instructions from the autonomouscontrol system 120, but also responds to commands from a manualoperator. For example, a manual operator may be able to specify an earthmoving routine for the EMV 130 to complete. Manual operators of theearth moving vehicle 130 may monitor the earth moving routine frominside of the earth moving vehicle 130 using the autonomous controlsystem 120 or remotely using an off-unit control system 180 located atthe work site or off-site in a remote location. Manual operation maytake the form of manual input to control input 140 or specific manualoperation only controls, such as an emergency stop. Sensor data isreceived by the autonomous control system 120 and assists in thecarrying out of those instructions, for example by modifying exactlywhat inputs are provided to the EMV 130 in order to achieve theinstructions to be accomplished as part of the earth moving routine.

In some implementations, the autonomous control system 120 trains andmaintains a set of machine learning models that help the autonomouscontrol system 120 to select actions and control signals for executionby the EMV 130. A machine learning model is an algorithm that isautomatically generated based on a set of training data (generally usingiterative methods) to predict one or more output values based on a setof input features. A training data set can contain many input featuresand, in some implementations, the corresponding output values. Dependingon the embodiment, the autonomous control system can use many differentmachine learning techniques, including supervised learning (usingtraining data labeled to include correct output values), unsupervisedlearning (using unlabeled training data), reinforcement learning, deeplearning or neural network algorithms, active learning, online learning,incremental learning, and other suitable techniques or algorithms. Theautonomous control system 120 can use machine learning models to, forexample, process sensor data (for example, to determine soil parametersfor material the EMV 130 is interacting with), optimize future actionsof the EMV 130 (by selecting an optimal tool path to perform an action),or to predict the response of an EMV 130 to a control signal (through agenerated calibration or control model).

In some implementations, the autonomous control system 120 uses onlinelearning techniques to train and update machine learning models. Anonline learning model (i.e. a machine learning model trained usingonline learning techniques) is continuously updated and refined as themodel is in use, using newly acquired training data as it received. Anonline learning model can be initially trained or instantiated usingbaseline training data (or using an assumed model) that is theniteratively refined each time a new data point is received or measured.For example, an online learning model predicting EMV 130 response to acontrol signal can be updated each time the EMV 130 executes a controlsignal (and the EMV response is measured using sensors 160). In someimplementations, online learning implementations update modelsincrementally based only on the newest training data points (or smallbatches of recent data points). In contrast, many other machine learningimplementations update models by fully retraining based on all availabledata (including the most recent training data). Using online learningcan allow an autonomous control system 120 to adapt its machine learningmodels to the current situation and environment of the EMV 130 duringoperation of the EMV 130. Using online learning techniques can eliminatedowntime and work interruptions resulting from taking systems offline toretrain (as may be necessary in non-online learning implementations).For example, the autonomous control system 120 can use online machinelearning to adapt a model for predicting soil parameters to the currentsoil or material on the work site as the EMV 130 is in operation. Theuse of online learning can also lead to quicker adaptation to currentconditions (relative to other machine learning methods) and more of afocus on the current conditions on the work site (as opposed to relianceon potentially stale static training data). In some implementations,online learning implementations have additional performance benefitssuch as less memory usage and reduced computational complexity whencompared to implementations not using online learning.

The autonomous control system 120 may also exchange information with theoff-unit control system 180 and/or other earth moving vehicles 130 (notshown) via the network 170. For example, an earth moving vehicle 130 maycommunicate recorded sensor data to a fleet of additional earth movingvehicles 130 on the work site. Similarly, through the network 170, thecontrol systems 120 and 180 may deliver data regarding a specific worksite to a central location from which the fleet of earth moving vehicles130 are stored.

The autonomous control system 120 may also process data received fromsensors 160. Processing can take sensor data in a “raw” format notdirectly usable by the autonomous control system 120 and converts itinto a form that is useful for analysis or decision making. For example,the autonomous control system 120 may synthesize data from the varioussensors into a real-time scan of the ground surface of the work sitearound the earth moving vehicle 130. For example, the autonomous controlsystem 120 can generate a digital environment map by fusing the pointclouds from various spatial sensors 130, stitching images from multipleimaging sensors 135, and registering the images and point cloudsrelative to each other or an external reference frame as provided byposition sensors 145 or other data sources. Processing may also includeup sampling, down sampling, interpolation, filtering, smoothing, orother related techniques to improve or refine the quality of receivedsensor data. The autonomous control system 120 will be discussed furtherin relation to FIG. 3 .

Off-unit Control System

In some implementations, the off-unit control system 180 includes asoftware architecture for supporting access and use of the earth movingsystem 100 and/or many different earth moving vehicles 130 throughnetwork 170. In some embodiments, operations or processing performed bythe autonomous control system 120 may also be performed similarly by theoff-unit control system 180. An off-unit control system 180 can includeservers, server clusters, cloud-based servers, or client devicescommunicatively connected to one or more EMV systems. Off-unit controlsystems 180 may include hardware located at the work site (but not on anEMV) as well as remote hardware located off site (such as at a controlcenter for the EMV system).

In some instances, the operation of the earth moving vehicle 130 ismonitored by a human operator riding in the EMV 130 or remotely throughthe off-unit control system 180. Human operators, when necessary, mayhalt or override an automated earth moving process and manually operate(either directly or through remote control) the earth moving vehicle 130in response to observations made regarding the features or theproperties of the work site. Monitoring by a human operator may includeremote oversight of all or part of an executing the earth moving routinethrough a graphical user interface (GUI) or other suitable means. Humanoperation of the earth moving vehicle 130 may also include manualcontrol of the earth moving vehicle 130 for set portions of an earthmoving routine or action or in situations where human operation isdetermined to be safer. A human operator can use controls of the EMV 130such as joysticks or other physical controls or off-board controls (suchas a “gamepad” style controller or computer keyboard) to control the EMV130. Additionally, when appropriate, human operators may override all ora part of the earth moving routine/operation currently being carried outby the EMV 130.

Network

The network 170 represents the various wired and wireless communicationpathways between the off-unit control system 180, off-unit sensors, andcomponents of the autonomous EMV system 110. In some implementations,the network 170 uses standard Internet communications technologiesand/or protocols. Thus, the network 170 can include links usingtechnologies such as Ethernet, IEEE 802.11, integrated services digitalnetwork (ISDN), asynchronous transfer mode (ATM), etc. Similarly, thenetworking protocols used on the network 150 can include thetransmission control protocol/Internet protocol (TCP/IP), the hypertexttransport protocol (HTTP), the simple mail transfer protocol (SMTP), thefile transfer protocol (FTP), etc. The data exchanged over the network170 can be represented using technologies and/or formats including thehypertext markup language (HTML), the extensible markup language (XML),etc. In addition, all or some links can be encrypted using conventionalencryption technologies such as the secure sockets layer (SSL), SecureHTTP (HTTPS) and/or virtual private networks (VPNs). In anotherembodiment, the entities can use custom and/or dedicated datacommunications technologies instead of, or in addition to, the onesdescribed above.

EMV Autonomous Control

As described above, an autonomous control system 120 of an autonomousEMV system 110 can autonomously determine desired actions (and thecorresponding control signals) for the EMV 130 to perform desiredearth-moving operations and routines. FIG. 3 is a block diagram of anautonomous control system of an earth moving vehicle, according to oneembodiment. The autonomous control system 130 of FIG. 3 includes asensor input module 310, an action planning module 320, an EMV interfacemodule 330, and a calibration module 340.

The sensor input module 310 can receive and process sensor data fromvarious sensors 160. In some implementations, the sensor input module310 gathers sensor data indicating the current state of the EMV 130 andthe surrounding environment. For example, the sensor input module 310gather sensor data indicating actuator and/or control surface 150positions as well as other parameters of the EMV, such as currentfuel/battery level, current speed, and the like.

In some implementations, the sensor input module 310 combines sensordata from multiple sensors 160 to generate data that can be further usedby the autonomous control system 120. For example, a sensor input module310 can generate environmental data including digital environment mapsof the area and terrain around the EMV 130 and soil parameters of soilor other material the EMV 130 is interacting with based on gatheredsensor data. “Soil parameters,” as used herein, are properties of amaterial surface an EMV 130 is driving over, manipulating with a tool155, or otherwise interacting with. The sensor input module 310 and dataprocessing steps (including the determination of soil parameters) willbe discussed further below.

The action planning module 320 selects actions (for example, earthmoving actions and/or routines) for the EMV 130 to execute. In someimplementations, the action planning module 320 selects earth movingactions and/or routines for the EMV 130 to perform based on a currentstate of the EMV 130, the surrounding environment, and/or the work sitein general (as determined using sensor data), and a current earth movingroutine. In some embodiments, the action planning module 320 receivesoutside instruction on earth moving routines, for example, when theactive earth moving routine is set manually by a human operator at thework site or through the off-unit control system 180.

Based on the gathered environmental data and the current state of theEMV 130, the action planning module can select and adapt an earth movingaction for the current situation. In some implementations, each earthmoving action is associated with a set of parameters chosen by theaction planning module 320 based on the current environmental conditionsand desired action by the EMV 130. For example, the parameters for anearth moving action using a tool 155 (for scooping or otherwisemanipulating material) can include a set of tool path parametersgoverning the path of the tool 155 as the action is performed. A toolpath, as used herein, represents the position of a tool 155 relative tothe EMV 130 and/or the environment over time as an earth moving actionis performed. In some embodiments, the action planning module 320generates target tool paths and/or target paths for other controlsurfaces 150 of the EMV 130 for each earth moving actions to beperformed by the EMV 130. For example, the autonomous control system 120may select a target path of the chassis of the EMV 130 for an operationin which the EMV 130 maneuvers to a target location.

In some embodiments, the action planning module 320 dynamically tracks,updates, or repeats earth moving actions, tool paths, and/or targetstates for the EMV 130 based on received sensor data. For example, asthe EMV 130 moves the earth moving tool 155 along a target tool path,soil friction may result in tool deviating from the target tool path,creating a different actual tool path travelled by the earth moving tool155. In one embodiment, the action planning module 320 can repeats thesame target tool path/earth moving action until the deviation betweenthe target tool path and the actual tool path is less than a thresholdor until some other outcome is achieved, such as a threshold amount ofmaterial is removed.

Similarly, when an obstruction, for example an obstacle or another earthmoving vehicle 130, is determined to be within the target tool path, theaction planning module 320 can store the geospatial location of theobstacle and, depending on the type of obstruction. avoid the obstacle,initiate an earth moving action 420 to remove or break up the detectedobstacle, or wait for the obstacle to be manually removed (or move outof the way on its own).

In some implementations, the EMV interface module 330 translates thetool paths and/or target control surface 150 states into a set ofcontrol signals which can be transmitted to the control input 140 of theEMV 130 for execution of the earth moving action. Depending on themethod of actuation of a control surface 150 (for example, hydraulic, ormechanically actuated using electrical or engine power) and the currentstate of the EMV 130, the magnitude of a control signal required toachieve a given effect may vary. For example, hydraulic actuators mayrequire different amounts of hydraulic pressure to manipulate a jointinto a desired position depending on the load applied to the joint andthe current position of the j oint. For example, raising an emptyexcavator bucket near to the EMV 130 requires much less force thanraising an excavator bucket full of material with the EMV’s arm fullyextended. Similarly, weather or temperature effects, wear, variancebetween EMVs 130, or environmental effects can all change the effect ofa given control signal on the position of the EMV 130. For example, ifthe ground is muddy or slippery, comparatively more forward/reverseinput may be needed to achieve a desired movement of the EMV 130 than ifthe ground was paved. In some embodiments, the EMV interface module 330accounts for these factors when determining control signals to reach thetarget state and/or tool path selected by the action planning module320.

An EMV interface module 330 can select control signals in real-timebased on the current state of the EMV 130 (which can be changingthroughout the performance of the earth moving action) and the responseof the EVM 130 to the current control signals. For example, if an issuedcontrol signal isn’t having the desired effect (for example, if acontrol surface 150 is not moving to the expected position) the EMVinterface module 330 can issue a modified control signal to achieve thedesired result or take other appropriate action (such as aborting theearth moving action or performing an emergency stop if the situationwarrants it).

In some embodiments, the EMV interface model 330 uses a calibration ofthe EMV 130 to account for some or all baseline differences between thecurrent EMV 130 and other similar EMVs of the same type or model. Due tomanufacturing variations, wear, or damage, one instance of an EMV 130may have slightly different control characteristics than another EMV(even if the EMVs 130 are of the same model and have the same controlsurface layout and nominal capabilities). A “calibration” or “EMVcalibration,” as used herein, is a set of parameters and/or uniquecharacteristics of a given EMV 130 allowing an autonomous control system120 to accurately control that specific EMV 130. For example, acalibration for an EMV 130 can include information on control surfacegeometry, relative strength of actuators for moving control surfaces150, and the like. In some implementations, a calibration is stored as aset of speed curves mapping control signal to actuation speed for eachcontrol surface 150 of the calibrated EMV 130. Each instance of an EMV130 can have a separately stored calibration tailored to that specificEMV 130.

In some embodiments, the calibration module 340 of the autonomouscontrol system 120 generates and maintains a calibration for theassociated EMV 130. The calibration module 340 can base calibration onpre-defined actions taken by the EMV 130 under controlled (or relativelycontrolled) conditions (herein, “calibration actions”). Each calibrationaction can include the application of one or more predetermined controlsignals to various control surfaces 150 in order to measure the effectsof the applied control signals. For example, a calibration action couldinvolve throttling the engine to increase the RPMs of the engine,putting the EMV 130 into gear, and measuring the resulting acceleration,resulting speed, and/or deceleration of the EMV 130 over time.Similarly, a different calibration action could include applying apredetermined control signal to a control surface 150 with a hydraulicactuator and measuring the resulting hydraulic pressure and/or themotion of the control surface 150 in response to the calibration controlsignal. In some implementations, the set of calibration actions includescalibration actions designed to rotate the body of the EMV 130 180degrees (both level and on an inclined surface) as well as calibrationactions for extending and retracting an excavator boom (or tool 155) ofthe EMV 130. One or more calibration actions can also be performedmanually, while the autonomous control system 120 monitors the controlsignals used to perform the calibration action.

FIG. 4 illustrates an example subdivision of a routine into operationsand control signals for an earth moving vehicle to perform, according toone embodiment. Environment 400 includes an example earth moving routine410 divided into a set of earth moving actions 420 each executed usingone or more control signals 450. As described above, earth movingroutines 410 can be associated with an objective (or task on the worksite) to be completed by the EMV 130. For example, work sitepreparation, digging or excavation, dumping or disposal of waste, andbackfill/compaction objectives can be associated with earth movingroutines 410. In some implementations, an earth moving routine 410 canbe associated with various parameters defining, for example, an area ofthe work site to excavate, the location of materials to move, thedesired depth of the excavation, and the like.

As described above, an autonomous control module 120 can select andperform earth moving actions 420 to accomplish the active earth movingroutine 410. In some implementations, each earth moving routine 410 (ortype of earth moving routine) is associated with a set of appropriateearth moving actions 420 for that specific routine 410. For example, anearth moving routine 410 for excavating a specific area can beassociated with earth moving actions 420 for scooping dirt from a givenpoint, maneuvering to an appropriate dump location, and dumping theexcavated dirt (among others).

Each earth moving action 420 can be associated with a suitable type ofEMV 130, a starting state (or states) of the EMV 130 for the earthmoving action 420 and a target ending state of the EMV. For example, ascooping operation can have a starting state of an EMV with an emptybucket within a certain range of the target scoop location and an endingstate of the EMV with a full bucket (and the arm in a neutral position.Earth moving actions 420 possible by an excavator are not necessarilythe same as the earth moving actions possible with a bulldozer, eventhough both EMVs 130 may be able to perform earth moving routines 410involving moving material around a work site. As described above, theaction planning module 330 can select earth moving actions 420 for anEMV 130 to perform (including selecting parameters for the earth movingactions 420). In FIG. 4 , the parameters of an earth moving action 420are influenced by a tool path model 430, EMV state data 440, andenvironmental data 445.

As described above, some earth moving actions 420 specify a target toolpath (representing the position of the tool 155 over time) that a tool155 will take when executing the operation 420. In some implementations,the action planning module 330 uses a tool path model to calculate orselect an optimal tool path for the operation 420 based on EMV statedata 440 about the current state and capabilities of the EMV 130 andenvironmental data 445 about the local environment of the EMV 130. Forexample, the tool path model 430 can take into account EMV state data440 such as the current position and orientation of the control surfaces150 of the EMV 130, any load on a control surface 150, and otherstatistics such as current fuel level etc. Similarly, the environmentaldata 445 can include processed sensor data such as a digital environmentmap of the immediate surroundings of the EMV 130 and calculated soilparameters (as described above) for a material the EMV 130 will beinteracting with using its tool 155. In some implementations, the toolpath model 430 is a machine learning model trained and maintained by theaction planning module 330. The tool path model 430 will be discussedfurther below.

In some implementations, each earth moving action 420 can be translatedinto a set of control signals 450 which perform the earth moving action420 when executed by the EMV 130. As described above, the EMV interfacemodule 330 can generate control signals 450 associated with variouscontrol surfaces of the EMV 130 based on their target positions in theearth moving action, a control model 460 modeling the response of eachcontrol surface 150 to control signal input, and an EMV calibration 470for the EMV 130. In some implementations, the control model 460 canpredict the response of a control surface 150 to a given control signalbased on the EMV state 440, environmental data 445, and the EMVcalibration 470. For example, the magnitude, timing, and otherparameters for a control signal 450 can be selected based on the controlmodel 460. The control model 460 and EMV calibration 470 will bediscussed further below.

Sensor Data Processing

As described above, the input module 310 receives and processes sensordata from various sensors 160 to generate EMV state and environmentaldata 440 and 445 that can be used to inform autonomous actions (such asearth moving actions 420) taken by the EMV 130.

For example, the sensor input module 310 can receive raw sensor datafrom a variety of sensors 160 (for example, multiple cameras mounted ondifferent parts of the EMV 130) and combine the received data into adigital environment map of the environment surrounding the EMV 130 usingsimultaneous localization and mapping (SLAM) techniques. Similarly, thesensor input module 310 may perform other processing operations, such asup sampling, down sampling, interpolation, filtering, smoothing, or thelike to improve the quality or useability of received sensor data. Thesensor input module 310, according to some embodiments, uses one or moremachine learning models to process received sensor data for use by theautonomous control system 120. As described above, the calculatedenvironmental data (including soil parameters) can then be incorporatedinto the selection and execution of EMV operations 420 as EMV state data440 or environmental data 445.

In some implementation the sensor input module 310 accesses, generates,or maintains a detailed map of the work site and/or the environmentsurrounding the EMV 130. The work site map can include image filesdescribing the geographic layout of the work site as a function oflocation within a coordinate space of the work site, with differentimages representing a dig location, fill location, an entry ramp, etc.In some implementations, the work site map includes information on thecurrent state of the work site, tasks the EMV 130 is intended to perform(for example, marking a volume for the EMV 130 to excavate), and thetarget state of the work site once EMV operations are completed. Forexample, a dig location may be characterized by a set of target volumedimensions which should be achieved upon the conclusion of an earthmoving routine. Additionally, the fill location (where excavatedmaterial is deposited) may be included in the work site map. Whenappropriate, the work site map can describe the location of fiducialsrepresenting technical pieces of equipment previously placed at the worksite such as stakes with active emitters and grade stakes.

The sensor input module 310 can, according to some embodiments, registerthe current position of the EMV 130 on the work site byaligning/registering the digital environment map (generated based onsensor data captured from the EMV 130) with the work site map(containing known information about the layout of the work site) usingthe known locations of fiducials and other locations within the worksite common to both representations. Position data from a positionsensor 145 such as a GPS or the boundaries of the work sites provided byboth representations may also be used by the sensor input module 310.

In some implementations, the sensor input model 310 measures or infersinformation about tools 155 of the EMV 130. For example, the sensorinput model 310 can use sensor data to determine the fill level of anexcavator bucket or other tool. The fill level of the tool describes thevolume of earth in or being manipulated by a tool 155 and can bedetermined by the autonomous control module 120 using many differentmethods. For example, fill level can be estimated mathematically byintegrating the depth of the leading edge beneath the ground surfaceover the distance traveled by the tool 155 beneath the ground surface,using a trained machine learning model, based on imagery or other datafrom sensors 160, or by inference using a kinematic force being exertedon the tool 155 by the material it’s traveling through and/or soilparameters of the material.

Commonly, soil parameters are calculated for a soil or dirt surfacebeing dug into or driven through by an EMV 130, but soil parameters canalso be calculated for suitable non-soil materials (such as constructionaggregate or gravel). Soil parameters can include a soil/materialresistance factor (a catch-all soil parameter measuring the relativeamount of force needed to move a tool 155 through the material relativeto other types of material) and/or more granular measures expressed inseparate parameters such as soil expansion, cohesion, density, tractionlevel, and the like. In some implementations, the sensor input module310 uses a trained soil parameter model to determine one or more soilparameters of material the EMV 130 is currently interacting with basedon sensor data measuring the current speed of a tool 155 through thematerial and a corresponding kinematic pressure on the tool 155. In someimplementations, the kinematic pressure represents not only the currentpressure/force on the tool 155, but the force exerted on the tool overtime. A kinematic pressure on a tool 155 can be determined using aspecialized force or pressure sensor built into the tool 155 or inferredbased on other sensor measurements or the state of the vehicle (such asan amount of control input or hydraulic pressure needed to maintain thetool 155 in its current position/state).

FIG. 5 is a block diagram of an example machine learning soil parametermodel for an earth moving vehicle, according to one embodiment. FIG. 5includes a soil parameter model 510 which identifies one or more soilparameters 530 of a material the EMV 130 is currently interacting withbased on inputs 515 including EMV state data 520 and/or current controlsignals 525 being sent to the EMV 130. In some implementations, the soilparameter model 510 can also take input of a baseline soil type (such as“high clay soil” or “sand”) which is set manually, based on the locationof the work site, or through other suitable methods. In the embodimentof FIG. 5 , the soil parameter model 510 is trained on soil parametertraining data 540 including a combination of baseline training data 545and online training data 550. As described above, online learning is amachine learning technique where the ML model is constantly updatedbased on newly received data points. When the autonomous control system120 uses the predicted soil parameters 530 for the EMV 130 to informinteractions with material (such as soil or earth), the observed resultsof these interactions become online data points 535 used as onlinetraining data 550 to update the soil parameter model 510. In someimplementations, the soil parameters determined by the soil parametermodel 510 can be used to inform EMV 130 path planning, volume generation(such as how much soil to scoop at once), and the like.

The autonomous control system 120 can use soil parameters to determinethe amount of power/force put into a given action (through changing themagnitude of a control signal or using alternate methods of actuatingthe tool) to adjust or maintain a desired tool speed, achievepenetration of the ground surface, and/or break out of the ground whileperforming a given action. Similarly, the autonomous control system 120can vary tool parameters other than tool speed and tool force, such asangle of attack and angle when lifting out of the ground based on soilparameters.

A soil parameter model 510 can be initialized by training the modelbased on baseline training data 545. In some embodiments, the baselinetraining data 545 contains a set of training data comprising data pointsgathered from a variety of different soil and material types includingexample model inputs and corresponding soil parameters for each type ofsoil. Once the soil parameter model 510 is initialized, the onlinemachine learning process can begin as the EMV 130 begins interactingwith material. In some embodiments, the autonomous control system 120stores a pre-trained baseline soil parameter model which is copied anytime a soil parameter model 510 is initialized.

In some implementations, the EMV state data 520 includes a current toolspeed for tool 155 currently interacting with the material. For example,the tool 155 can be an excavator/loader bucket or bulldozer blade whichis scooping through or pushing material. The soil parameter model 510can also input one or more current control signals 525 the EMV 130 isexecuting. EMV control signals can be used to estimate an amount ofkinematic pressure exerted by the tool on the material (andcorrespondingly, by the material on the tool). For example, the currentamount of force exerted by a hydraulic cylinder can be regulated by acontrol signal to the corresponding control surface 150. In otherimplementations, the current kinematic pressure on a tool is directlycollected or inferred as part of gathered sensor data. For example, sometools 155/control surfaces 150 may have inbuilt pressure sensors tomeasure how much force is being exerted on the tool 155 or controlsurface 150. In some implementations, the soil parameter model 510 canalso take input of kinematic pressure over time. For example, arelatively constant kinematic pressure may indicate a uniform material,while kinematic pressure that spikes and gives way intermittently mayindicate a rocky material that exhibits strong resistance at first, thenbreaks free all at once.

In some embodiments, each suitable interaction between the tool 155 ofthe EMV 130 and soil or other material is measured and used as an onlinedata point 535. Online data points 535 can be used to update the soilparameter model 510 without requiring full retraining of the model. Inthese implementations, the effective training set grows over time as theamount of online training data 550 taken into account by the model 510grows, but the soil parameter model 510 does not need to be trained onthe full training data set 540 (including all baseline data 545 andonline data 550) in one batch. In some embodiments, online data points535 are discarded soon after being used to update the soil parametermodel 510. Alternatively, online data points 535 can be archived and/orsent to the off-unit control system 180 for later use (for example, forinclusion in the baseline training data 545 for future EMVs 130).

In some embodiments, the soil parameter model 510 is re-initializedperiodically so that the soil parameter predictions are adapted tocurrent soil conditions. For example, the soil parameter model 510 maybe re-initialized (reset to use only baseline training data 545) at thestart of each day and/or work session to account for soil conditionchanges overnight (or since the EMV 130 was last used). For example,overnight rain or other weather events could alter the soil/materialproperties enough that using a soil parameter model 510 trained on theprevious day could be counterproductive (producing worse results orimpeding the soil parameter model 510 from quickly adapting to new worksite conditions).

Tool Path Optimization

As described above, the action planning module 320 selects actions (forexample, earth moving actions and/or routines) for the EMV 130 toexecute based on the gathered environmental data and the current stateof the EMV 130. Earth moving actions 420 can be associated with a set ofparameters including tool path parameters governing the path of the tool155 as the action is performed. As described above, a tool pathrepresents the position of a tool 155 relative to the EMV 130 and/or theenvironment over time. A target tool path can include target positionsof one or more control surfaces 150 at various points in the tool path.

An EMV 130 generally has many control surfaces 150, including controlsurfaces associated with a tool 155 of the EMV 130. FIG. 6 illustratesexample placement of control surfaces for an excavator, according to oneembodiment. The environment 600 includes an excavator 610 with a bucket615 and multiple joints, including a cab rotation joint 650 and an armassembly including a boom joint 620, arm joint 630, and bucket joint640. Additionally, some joints highlighted in FIG. 6 are associated witha corresponding hydraulic actuator, such as the boom actuator 625 forthe boom joint 620, arm actuator 635 for the arm joint 630, and bucketactuator 645 for the bucket joint 640. The excavator 610 may haveadditional control surfaces not depicted in FIG. 6 (for example, controlsurfaces 150 related to skid steer control of the tracks of theexcavator 610).

FIG. 6 shows several control surfaces 150 of an example excavator 610(the boom joint 620, arm joint 630, and bucket joint 640) and thecontrol mechanisms that actuate the control surfaces 150 (the boomactuator 625, arm actuator 635, and bucket actuator 645, respectively).In some implementations, the bucket 615 is a tool 155 of the excavator,and the boom joint 620, arm joint 630, and bucket joint 640 are controlsurfaces 150 which allow the tool 155/bucket 615 to be manipulatedthrough a desired tool path through control signals controlling thecorresponding actuators 625, 635, and 645.

FIG. 7 illustrates example tool path parameters of a scooping operationfor an excavator, according to one embodiment. The environment 700 ofFIG. 7 includes an excavator 710 (a type of EMV 130) performing ascooping earth moving action 420 using its bucket 715 (a tool 155).Example tool path parameters of the scooping operation include the scoopstart distance 720, angle of attack 730, and scoop depth 740. The scoopstart distance 720 controls how far away from the excavator 710 chassisthe bucket 715 will initially enter the ground surface (a shorter scoopstart distance 720 provides greater leverage for the excavator 710 butmay result in more maneuvering the chassis on consecutive scoops).Similarly, the angle of attack 730 controls the angle of the bucket 715when it enters and travels through the ground and the scoop depth 740measures how far the bucket 715 will penetrate below the surface of thematerial during the operation. Additionally, an excavator scooping earthmoving action can have additional tool path parameters, such as a targetspeed of the scooping motion, a state of the tool, and a target bucket715 fill amount for the scoop.

Depending on the parameter and earth moving action 420, tool pathparameters can represent specific metrics (such as an angle or distanceat a certain point in the tool path), can be binary values (for example,if ground stabilizers are deployed during the operation 420 or if ajackhammer tool 155 is active or not), or may be associated with one ormore archetypal tool paths which are blended based on the parametervalue. For example, the scoop depth parameter 740 can be associated withtool paths for a “deep scoop” and a “shallow scoop” which are blendedbased on the scoop depth parameter 740 to determine the target tool pathfor the operation 420.

The selection of tool path parameters can meaningfully affect theefficacy of the earth moving action 420. For example, the angle ofattack 730 of the bucket 715 beneath the ground surface during anexcavation action can be adjusted to reduce the rate at which the earthmoving tool 155 collects excavated earth. The rate of excavation can beat its highest when the bucket 715 is angled perpendicular to the flatground surface, and the lowest when the bucket 715 is angled parallel tothe flat ground surface. Additionally, at lower speeds, the EMV 130 isgenerally often better able to maintain the optimal bucket angle ofattack 730 for excavating earth.

As described above, the action planning module 320 can generate one ormore candidate tool paths for performing a selected earth moving action420. In implementations where the work site is represented in thedigital terrain model as a coordinate space, for example theimplementations described above, a target tool path includes a set ofcoordinates within the coordinate space. Target tool paths may describea variety of shapes representing a variety of earth moving techniques,for example substantially rectangular pathways in two dimensions,substantially triangular pathways in two dimensions, hyperrectangularpathways in three dimensions, hyperrectangular pathways in threedimensions, elliptic pathways in two dimensions, hyperelliptic pathwaysin three dimensions, or curved lines along the plane of the groundsurface. An earth moving action 420 can involve the execution ofmultiple tool paths, such as in the case of repetitive actions, actionsthat require coordination between multiple tools 155 or EMVs 130, orcomplex actions with multiple stages.

For example, for holes of greater volumes or requiring a gradedexcavation, action planning module 320 may generate multiple target toolpaths gradually building the size and/or depth of the excavation. As aresult, a target tool path may represent instructions for excavatingonly a fraction of the total volume of the excavation. In someimplementations, the last tool path used at the conclusion of theexcavation is a finish tool path, which digs minimal to no volume andprimarily evens the surface of the bottom of the excavation.

In some embodiments, the action planning module 320 generates one ormore candidate tool paths which are evaluated and optimized to selectthe target tool path for the earth moving action 420. In someimplementations, the autonomous control system 120 maintains and updatesa machine learning model trained to determine an optimal tool path for agiven operation 420 based on the current state of the EMV 130 and thework site. FIG. 8 is a block diagram of an example machine learning toolpath model for an earth moving vehicle, according to one embodiment.FIG. 8 includes a tool path model 810 which identifies an optimal toolpath 830 of a material the EMV 130 is currently interacting with basedon inputs 815 including sensor data 820, a target action 822, and a setof candidate tool paths 824. In the embodiment of FIG. 8 , the tool pathmodel 810 is trained on tool path training data 840 including acombination of baseline training data 845 and online training data 850.As described above, online learning is a machine learning techniquewhere the ML model is constantly updated based on new online data points835 created based on the observed results of the optimal tool paths 830previously selected by the tool path model 810. In some implementations,the autonomous control system 120 maintains a set of tool path models810, each trained to optimize tool paths for different types of action.For example, the autonomous control system 120 of an excavator 710 canmaintain a tool path model 810 trained to optimize scooping actions anda separate tool path model 810 for dumping actions.

Depending on the implementation and the target action 822, the tool pathmodel 810 can be trained to optimize for one or more action conditionsin the selection of a tool path. As used herein, an action condition isa metric or measurement that the tool path model 810 is configured tooptimize for. For example, action conditions can include efficacy of thetarget action, fuel/energy efficiency of the tool path, speed of thetool path, amount of force required to complete the tool path, volume ofmaterial affected by the tool path, or amount of motion required of aset of control surfaces 150 (such as preferring tool paths that do notrequire moving the chassis of the EMV 130). Similarly, an actioncondition can optimize for the number of repetitions of a repeatedaction (for example, the number of scoops needed to complete a targetexcavation). To optimize the tool path, the tool path model 810 canselect values for one or more tool path parameters as described above inrelation to FIG. 7 .

Like the soil parameter model 510, the tool path model can beinitialized by training on baseline training data 845. In someembodiments, the baseline training data 845 contains a set of trainingdata comprising data points gathered from a different times one or moretarget actions were performed in different EMV states and environmentalconditions (as measured by the sensor data 820). Once the tool pathmodel 810 is initialized, the online machine learning process can beginas the EMV 130 performs the target action 822.

In some implementations, the sensor data 820 includes state data for theEMV 130 and environmental data about the surrounding environment of theEMV 130. For example, the angle of the EMV 130, the current fuel levelof the EMV 130, or calculated soil parameters can affect tool pathselection. For example, a high soil resistance factor can influence toolpaths to account for greater force needed to accomplish the targetaction 822 through high resistance material (for example, performingscooping actions closer to the chassis of the EMV 130 or choosing a morevertical angle of attack).

The set of candidate tool paths 824 can include a plurality of candidatetool paths that can be separately evaluated and compared by the toolpath model 810 and/or one or more template tool paths that the tool pathmodel 810 can select from, modify, or blend together (depending on theimplementation) to arrive at an optimal tool path 830 for a givensituation. Each candidate tool path 824 can be associated with one ormore calculated statistics, such as a baseline time to complete the toolpath (for example, based on the distance the tool/EMV 130 needs totravel to complete the tool path). In some implementations, the actionplanning module 320 can procedurally generate a set of candidate toolpaths can be generated by blending between a set of template tool pathsand/or randomly varying one or more tool path parameters of a tool path(such as attack angle 730 for a scoop action). In other implementations,the action planning module 320 iteratively tunes a single candidate toolpath 824 using the tool path model 810. A candidate tool path can begenerated based on a template tool path (a generic tool path for a givenearth moving action) or based on one or more recent tool paths used forthat earth moving action. For example, in the case of a repetitiveaction a candidate tool path can be generated by averaging (or otherwisecombining) several of the most recent tool paths used for that action.Previous tool paths can be combined by averaging key frames of the toolpath, tool path parameters, and the like (depending on the embodiment).A candidate tool path can be generated in this way using a weightedaverage (for example, heavily weighting the tool path for the mostrecent use of that action by the EMV 130) or another suitablestatistical method.

For example, to determine an optimal tool path 830 from a set ofcandidate tool paths 824, the action planning module 320 can apply thetool path model 810 to each of the candidate tool paths 824 to determinea score or metric of each candidate tool path 824 with respect to one ormore action conditions based on the target action 822 and sensor data820. Once each candidate tool path 824 is evaluated by the tool pathmodel 810 (with respect to one or more action conditions, as describedabove), the tool path model 810 and/or action planning module 320 ranksthe set of candidate tool paths based on the to arrive at the optimaltool path 830 for the earth moving action 420. In other implementations,the tool path model 810 can be used to evaluate a candidate tool path824 against one or more threshold scores. If the candidate tool path 824does not meet the threshold criteria, the candidate tool path 824 can beiteratively refined to improve the evaluation by the tool path model 810(and then re-evaluated).

In some embodiments, each time the EMV 130 performs the target action822 sensor data is gathered and used as an online data point 835 forthat action. As described above, online data points 835 can be used toupdate the tool path model 810 without requiring full retraining. As theEMV 130 executes the earth moving action 420 using the selected optimaltool path 830, the action conditions of the selected tool path 830 canbe measured (using sensors 160 etc.) and used as an online data point835 to update the tool path model 810. Similar to the soil parametermodel 510, observed online data points 835 can be discarded soon afterbeing used or archived for later use (for example, for inclusion in thebaseline training data 845 for future EMVs 130). EMV Control Model

As described above, a control surface 150 of an EMV 130 may reactdifferently to a received control signal based on the current state andenvironmental conditions of the EMV. Depending on the type of controlinput 140 and the characteristics of the specific control surface 150getting predictable results from a selected control signal (which can beimportant to safe and effective autonomous operation) can be nontrivial.

In some embodiments, control signals applied through a mechanicalcontrol input (or a retrofitted manual control input) have limitedprecision. For example, control signals applied to mechanical controlsfor hydraulic control surfaces 150 can be restricted to a simpleleft/right or on/off scale adapted from manual controls for that controlsurface 150.

Similarly, certain control surfaces 150 (especially those that havesignificant weight on them) may have a fixed (or not directlycontrollable) deceleration rate due to inertia and the way the controlsurface 150 was constructed. For example, the cab/chassis rotation joint650 of the excavator 615 (i.e. the joint that allows the tracks ofexcavator 615 to rotate independently of the main body of the excavator)can exhibit substantial inertia once moving. In this case, cab of theexcavator 615 cab will continue rotating due to inertia after thecontrol signal instructing the cab/chassis rotation joint 650 to rotatehas ended (or even been reversed). In some implementations, the EMVinterface module 330 uses one or more machine learning models (controlmodels 460) to translate a desired action or target position of acontrol surface 150 into one or more control signals of the proper typeand magnitude to achieve the desired end state in the currentenvironment. For example, the EMV interface module 330 can use thecontrol model 460 to achieve a desired cab rotation by sending a controlsignal to the cab/chassis rotation joint 550 such that the cab naturallyslows to a stop at the desired position.

In some implementations, the EMV interface module 330 can dynamicallyadjust control signals to maintain a target tool path or achieve atarget state of one or more control surfaces 150. For example, adjustingthe angle of a tool 155 to a shallower depth or angle to maintain adesired tool speed based on soil parameter data or deviation from anexpected tool speed. Similarly, in some implementations the EMVinterface module 330 times control signals to account for hydraulicresponse delay or adjusts sequencing of control signals based on alearned total hydraulic capacity of the EMV 130 (accounting for orpreventing losing hydraulic power when trying to move multiple hydrauliccircuits at the same time).

FIG. 9 is a block diagram of an example machine learning control modelfor an earth moving vehicle, according to one embodiment. FIG. 9includes a control model 910 which selects a control signal 930 toachieve a target state 922 of the EMV 130 based on inputs 915 includingsensor data 920 measuring the actual state of one or more controlsurfaces 150, the target state 922 of one or more control surfaces 150,and, in some implementations, an EMV calibration 924. In the embodimentof FIG. 9 , the calibration model 910 is trained on control modeltraining data 940 including a combination of baseline training data 945and online training data 950. As described above, online learning is amachine learning technique where the ML model is constantly updatedbased on new online data points 935 created based on the observedresults of the application of the control signals 930 previouslygenerated by the control model 910. In some implementations, theautonomous control system 120 maintains a set of control models 910,each trained to determine control signals 930 for different types ofearth moving action. When applied in practice, the control model 910 cangenerate control signals 930 that can be applied to one or more controlsurfaces 150 to achieve the target state 922.

In some implementations, the sensor data 920 includes state data for theEMV 130 and environmental data about the surrounding environment of theEMV 130 (such as the angle of the EMV 130 or the load on the EMV 130 canaffect control signal selection). Similarly, the target state 922 of theEMV can be used as an input for the control model 910. For example, thecurrent and target states of the EMV 130 can be compared as part ofdetermining the types and magnitudes of control signal 930 to apply.Additionally, the EMV calibration 924 can inform how much the EMV 130will react to a given magnitude of control signal. the EMV calibration924 will be discussed further below.

Like the soil parameter model 510, the control model can be initializedby training on baseline training data 945. In some embodiments, thebaseline training data 945 contains a set of training data comprisingdata points gathered from a different times one or more target actionswere performed in different EMV states and environmental conditions (asmeasured by the sensor data 920). Once the control model 910 isinitialized, the online machine learning process can begin as the EMV130 performs the earth moving actions.

In some embodiments, each time the EMV 130 performs an earth movingaction based on a control signal 930 generated from the control model910, the result of the control signal 930 can be measured and (alongwith the target state 922 associated with that earth moving action) usedas an online data point 935 for that earth moving action. Each onlinedata point 935 can include the initial state of the EMV 130, the appliedcontrol signal (and target state for that control signal) and the actualstate of the EMV 130 after the control signal was executed. As describedabove, online data points 935 can be used to update the control model910 without fully retraining the model. Similar to the soil parametermodel 510, observed online data points 935 can be discarded soon afterbeing used or archived for later use (for example, for inclusion in thebaseline training data 945 for future EMVs 130).

EMV Calibration Model

As described above, the autonomous control system 120 can use acalibration to account for some or all differences between the currentEMV 130 and other similar EMVs of the same type or model (for example,due to manufacturing variations, wear, or damage). An EMV calibrationcan include information on control surface geometry and the relativestrength of actuators for moving control surfaces 150 and may be storedas a set of speed curves mapping control signal to actuation speed forcontrol surfaces 150 of the calibrated EMV 130.

In some implementations, the calibration process is periodicallyrevisited as the calibration drifts out of line with real worldperformance (indicating wear on the EMV 130 to the extent that it altersbaseline performance of the EMV 130). The calibration module 340 canadjust the calibration for the wear, re-do the calibration process,and/or notify the off-unit control system 180 that maintenece needs tobe performed depending on the severity of the performance drift.

The calibration module 340 can generate and maintain EMV calibrations byinstructing the EMV 130 to perform a set of specific calibrationactions. Each type of calibration action can be associated with a set ofpredetermined control signals for various control surfaces 150. Thecalibration for the EMV 130 can be determined based on the EMV 130′sresponse to each of the calibration actions in a neutral setting (forexample, how much each control surface 150 moved in response to thecalibration action). In some implementation, the calibration processcontinues until the EMV calibration achieves a threshold level ofprecision (such as when the calibration accurately predicts the outcomeof the calibration actions).

In some implementations, the autonomous control system 120 generates anEMV calibrating by training a machine learning model based on the EMV130′s performance of one or more calibration actions. FIG. 10 is a blockdiagram of an example machine learning calibration model for an earthmoving vehicle, according to one embodiment. FIG. 10 shows an EMVcalibration 1010 which is iteratively refined as the EMV 130 performs aseries of calibration actions 1055. The EMV calibration 1010 can map aninput 1015 control signal 1020 to a predicted EMV response 1030 (forexample, in the form of a speed curve mapping control magnitude toactuation speed of one or more control surfaces 150).

In the embodiment of FIG. 10 , the EMV calibration 1010 is trained oncalibration training data 1040 including a set of stored baselinecalibration actions 1045 and a set of online calibration actions 1050performed by the EMV 130. As described above, online learning models aretrained iteratively as new data points are received and factored intothe model. For example, each calibration action 1055 of the set ofonline calibration actions 1050 can include a control signal 1060 (theinput to the EMV calibration 1010) and an observed result 1065 of one ormore control surfaces 150 responding to the control signal 1060 (thetarget output of the EMV calibration 1010). In some implementations, theEMV calibration 1010 is trained iteratively as the EMV 130 runs througha calibration routine of calibration actions 1055. As described above,the EMV calibration 1010 can be iteratively refined using onlinelearning without fully retraining the model after each calibrationaction 1055. Similar to the soil parameter model 510, observedcalibration actions 1055 can be discarded soon after being used orarchived for later use (for example, for inclusion in the baselinecalibration data 1045 for future EMVs 130).

In some implementations, the EMV calibration 1010 is locked (and notupdated) unless the EMV 130 performing one of a set of specificcalibration actions 1055 or is in a calibration mode. In someimplementations, calibration actions 1055 are designed to isolate andtest the responses of control surfaces 150 to various magnitudes ofcontrol signals. An EMV calibration 1010 can be initialized usingbaseline calibration data 1045 containing, in some implementations, datafrom another instance of the same type of EMV 130. In someimplementations, the baseline calibration data 1045 includes apre-trained baseline calibration that can be adapted to the specific EMV130 through online learning. As described above the calibration processcan be repeated (starting from the baseline calibration) or refined (byperforming more calibration actions) when performance drift from thebaseline calibration is detected.

EMV Autonomous Control Routines

As described above, the autonomous control system 120 maintains a soilparameter model trained to determine soil parameters of material the EMVis interacting with. FIG. 11 is a flowchart describing a process fordetermining soil parameters of soil interacted with by an earth movingvehicle using a machine learning soil parameter model, according to oneembodiment. The process 1100 begins when an EMV takes 1110 an actionthat involves a tool of the EMV moving through or otherwise interactingwith soil (or other similar material as described above). For example,scooping or dumping soil as part of an excavation. The autonomouscontrol system associated with the EMV can then identify 1120 a currentstate of the tool (including a current speed/velocity of the tool andkinematic pressure currently exerted on the tool) based on sensor data.To predict soil parameter, the autonomous control system accesses 1130an online learning model trained to determine soil parameters of thematerial based on the current speed and kinematic pressure exerted onthe tool (or other information about the state of the tool). Using theidentified tool state information as an input to the online learningmodel, the autonomous control system determines 1140 a set of soilparameters for the material the EMV is currently interacting with.Finally, the autonomous control system can use the determined soilparameters to inform future actions by the EMV. For example, theautonomous control system can select 1150 an optimal tool speed, toolpath, or other tool property based on a determined soil resistancefactor of the material being interacted with.

FIG. 12 is a flowchart describing a process for selecting a tool pathfor an earth moving vehicle using a machine learning tool path model,according to one embodiment. The process 1200 begins when an autonomouscontrol system selects 1210 an earth moving action for an EMV to performusing a tool. Then, the autonomous control system accesses 1220 anonline learning model trained to predict the performance of a tool pathwith respect to a set of action conditions. As described above, theaction condition is a metric used to evaluate the success of the targetaction (such as the predicted speed or fuel efficiency of the action).Based, for example, on the type of target action and the current stateof the EMV, the autonomous control system identifies 1230 a set ofcandidate tool paths for performing the target action. The autonomouscontrol system then selects an optimal tool path for the currentsituation by applying 1240 the retrieved online learning model to thecandidate tool paths. For example, the autonomous control system canevaluate each candidate tool path using the tool path model and selectthe highest ranked candidate tool path with respect to the set of actionconditions. The autonomous control system can then instruct the EMV toexecute the target action using the selected optimal tool path. Theactual result of the execution of the target action is then observed1260 by measuring the set of action conditions based on collected sensordata. For example, the speed of the earth moving action can becalculated by measuring the amount of time the EMV takes to reach an endstate of the earth moving action. In some implementations, the onlinelearning model is updated 1270 based on the observed results of theearth moving action, as described above.

FIG. 13 is a flowchart describing a process for controlling an earthmoving vehicle using a machine learning control model, according to oneembodiment. The process 1300 begins when an autonomous control systemselects 1310 a desired future state for a portion of the EMV includingone or more control surfaces. For example, a tool path for an earthmoving action can include one or more target states for control surfacesof the EMV, as described above. Additionally, the autonomous controlsystem then determines 1320 a current state of the EMV including thecurrent state of the portion of the EMV, for example by measuring thecurrent positions of each control surface of the EMV. Then, theautonomous control system accesses 1330 an online learning model trainedto determine what control signal to apply one or more control surfacesto achieve the desired future state. For example, the autonomous controlsystem can use the control model to determine how much force to apply toachieve a target speed of the EMV or how long to apply an input toachieve a desired position of the EMV’s arm. Then, the autonomouscontrol system uses the control model to select 1340 a set of controlsignals and/or the associated timings to instruct the EMV to move to thedesired future state. The autonomous control system the executes 1350the selected control signals by sending them to a control input 140 ofthe EMV for execution. As the EMV executes the selected control signals,the autonomous control system monitors 1360 one or more updated statesof the portion of the EMV. In some implementations, the autonomouscontrol system updates 1370 the control model based on the observedupdated state and the selected control signals, as described above.

FIG. 14 is a flowchart describing a process for calibrating an earthmoving vehicle using a machine learning calibration model, according toone embodiment. The process 1400 begins when an autonomous controlsystem selects 1410 a calibration control signal for a control surfaceof an EMV as part of a calibration process. As described above, the EMVcan run through a calibration process periodically or based on certaintrigger conditions being met.

Then, the autonomous control system accesses 1420 a calibration modelfor the EMV including an online learning model trained based on previouscalibration actions taken by the EMV (or similar EMVs). As describedabove, the autonomous control system can predict 1430 a result state ofthe EMV in response to the calibration control signal using thecalibration model. The autonomous control system the executes 1350 thecalibration control signal by sending it to a control input 140 of theEMV for execution and measures 1450 the state of the control surface asthe control signal is executed. In some implementations, the autonomouscontrol system updates 1460 the control model based on the observedupdated state and the calibration control signal, as described above.

General Computer Structure

As described herein, the autonomous control system 120 and off-unitcontrol system 180 may include one or more generic or special purposecomputers. A simplified example of the components of an example computeraccording to one embodiment is illustrated in FIG. 15 .

FIG. 15 is a high-level block diagram illustrating physical componentsof an example off-unit control system 180, according to one embodiment.Illustrated is a chipset 1505 coupled to at least one processor 1510.Coupled to the chipset 1505 is volatile memory 1515, a network adapter1520, an input/output (I/O) device(s) 1525, and a storage device 1530representing a non-volatile memory. In one implementation, thefunctionality of the chipset 1505 is provided by a memory controller1535 and an I/O controller 1540. In another embodiment, the memory 1515is coupled directly to the processor 1510 instead of the chipset 1505.In some embodiments, memory 1515 includes high-speed random accessmemory (RAM), such as DRAM, SRAM, DDR RAM or other random access solidstate memory devices.

The storage device 1530 is any non-transitory computer-readable storagemedium, such as a hard drive, compact disk read-only memory (CD-ROM),DVD, or a solid-state memory device. The memory 1515 holds instructionsand data used by the processor 1510. The I/O controller 1540 is coupledto receive input from the EMV 130 and sensors 160, as described in FIG.1 , and displays data using the I/O devices 1545. The I/O device 1545may be a touch input surface (capacitive or otherwise), a mouse, trackball, or other type of pointing device, a keyboard, or another form ofinput device. The network adapter 1520 couples the off- unit controlsystem 180 to the network 170.

As is known in the art, a computer can have different and/or othercomponents than those shown in FIG. 2 . In addition, the computer canlack certain illustrated components. In one embodiment, a computeracting as server may lack a dedicated I/O device 345. Moreover, thestorage device 1530 can be local and/or remote from the computer (suchas embodied within a storage area network (SAN)), and, in oneembodiment, the storage device 1530 is not a CD-ROM device or a DVDdevice. Generally, the exact physical components used in the autonomouscontrol system 120 and off-unit control system 180 will vary. Forexample, the autonomous control system 120 may be communicativelycoupled to the controller 150 and sensor assembly 110 differently thanthe off-unit control system 180.

Typically, the off-unit control system 180 will be a server class systemthat uses powerful processors, large memory, and faster networkcomponents compared to the autonomous control system 120, however thisis not necessarily the case. Such a server computer typically has largesecondary storage, for example, using a RAID (redundant array ofindependent disks) array and/or by establishing a relationship with anindependent content delivery network (CDN) contracted to store, exchangeand transmit data such as the asthma notifications contemplated above.Additionally, the computing system includes an operating system, forexample, a UNIX operating system, LINUX operating system, or a WINDOWSoperating system. The operating system manages the hardware and softwareresources of the off-unit control system 180 and also provides variousservices, for example, process management, input/output of data,management of peripheral devices, and so on. The operating systemprovides various functions for managing files stored on a device, forexample, creating a new file, moving or copying files, transferringfiles to a remote system, and so on.

As is known in the art, the computer is adapted to execute computerprogram modules for providing functionality described herein. A modulecan be implemented in hardware, firmware, and/or software. In oneembodiment, program modules are stored on the storage device 1530,loaded into the memory 1515, and executed by the processor 1510.

Additional Considerations

The foregoing description of the embodiments of the invention has beenpresented for the purpose of illustration; it is not intended to beexhaustive or to limit the invention to the precise forms disclosed.Persons skilled in the relevant art can appreciate that manymodifications and variations are possible in light of the abovedisclosure.

Some portions of this description describe the embodiments of theinvention in terms of algorithms and symbolic representations ofoperations on information. These algorithmic descriptions andrepresentations are commonly used by those skilled in the dataprocessing arts to convey the substance of their work effectively toothers skilled in the art. These operations, while describedfunctionally, computationally, or logically, are understood to beimplemented by computer programs or equivalent electrical circuits,microcode, or the like. Furthermore, it has also proven convenient attimes, to refer to these arrangements of operations as modules, withoutloss of generality. The described operations and their associatedmodules may be embodied in software, firmware, hardware, or anycombinations thereof.

Any of the steps, operations, or processes described herein may beperformed or implemented with one or more hardware or software modules,alone or in combination with other devices. In one embodiment, asoftware module is implemented with a computer program productcomprising a computer-readable medium containing computer program code,which can be executed by a computer processor for performing any or allof the steps, operations, or processes described.

Embodiments of the invention may also relate to an apparatus forperforming the operations herein. This apparatus may be speciallyconstructed for the required purposes, and/or it may comprise ageneral-purpose computing device selectively activated or reconfiguredby a computer program stored in the computer. Such a computer programmay be stored in a non-transitory, tangible computer readable storagemedium, or any type of media suitable for storing electronicinstructions, which may be coupled to a computer system bus.Furthermore, any computing systems referred to in the specification mayinclude a single processor or may be architectures employing multipleprocessor designs for increased computing capability.

Embodiments of the invention may also relate to a product that isproduced by a computing process described herein. Such a product maycomprise information resulting from a computing process, where theinformation is stored on a non-transitory, tangible computer readablestorage medium and may include any embodiment of a computer programproduct or other data combination described herein.

Finally, the language used in the specification has been principallyselected for readability and instructional purposes, and it may not havebeen selected to delineate or circumscribe the inventive subject matter.It is therefore intended that the scope of the invention be limited notby this detailed description, but rather by any claims that issue on anapplication based hereon. Accordingly, the disclosure of the embodimentsof the invention is intended to be illustrative, but not limiting, ofthe scope of the invention, which is set forth in the following claims.

What is claimed is:
 1. A method comprising: training, by a trainingsystem, a machine-learned model based on a training set of data thatdescribes movement instructions provided to one or more autonomousoff-road vehicles (AOVs) and resulting AOV movements, themachine-learned model configured to predict an AOV movement resultingfrom a movement instruction; determining, by the training system, thatan actual AOV movement for a first movement instruction is differentthan a predicted AOV movement by a threshold movement amount; andretraining, by the training system, the machine-learned model such thata difference between a subsequent actual AOV movement and the intendedAOV movement is less than the threshold movement amount.
 2. The methodof claim 1, wherein the machine-learned model is further configured togenerate a control signal for actuating a control surface of the AOVbased on the predicted AOV movement resulting from the movementinstruction.
 3. The method of claim 2, wherein the actual AOV movementis measured based on a state of the control surface at a time when thecontrol signal is executed.
 4. The method of claim 1, wherein thetraining system is configured to compare the actual AOV movement to thepredicted AOV movement periodically.
 5. The method of claim 1, whereinthe machine-learned model is further trained based on baseline trainingdata gathered from a set of additional AOVs similar to the AOV.
 6. Themethod of claim 1, wherein the predicted AOV movement comprises apredicted movement of a tool of the AOV.
 7. The method of claim 1,wherein retraining the machine-learned model comprises: updating, byonline learning techniques, the machine-learned model based on theactual movement corresponding to the first movement instruction.
 8. Themethod of claim 1, wherein the actual AOV movement is compared to thepredicted AOV movement responsive to the AOV switching between anautonomous operation mode and a manual operation mode.
 9. The method ofclaim 1, wherein the actual AOV movement corresponding to the firstmovement instruction is determined by measuring a set of actionconditions based on collected sensor data.
 10. A non-transitorycomputer-readable storage medium storing executable instructions that,when executed by a processor, cause the processor to: train, by atraining system, a machine-learned model based on a training set of datathat describes movement instructions provided to one or more autonomousoff-road vehicles (AOVs) and resulting AOV movements, themachine-learned model configured to predict an AOV movement resultingfrom a movement instruction; determine, by the training system, that anactual AOV movement for a first movement instruction is different than apredicted AOV movement by a threshold movement amount; and retrain, bythe training system, the machine-learned model such that a differencebetween a subsequent actual AOV movement and the intended AOV movementis less than the threshold movement amount.
 11. The non-transitorycomputer-readable storage medium of claim 10, wherein themachine-learned model is further configured to generate a control signalfor actuating a control surface of the AOV based on the predicted AOVmovement resulting from the movement instruction.
 12. The non-transitorycomputer-readable storage medium of claim 11, wherein the actual AOVmovement is measured based on a state of the control surface at a timewhen the control signal is executed.
 13. The non-transitorycomputer-readable storage medium of claim 10, wherein the trainingsystem is configured to compare the actual AOV movement to the predictedAOV movement periodically.
 14. The non-transitory computer-readablestorage medium of claim 10, wherein the machine-learned model is furthertrained based on baseline training data gathered from a set ofadditional AOVs similar to the AOV.
 15. The non-transitorycomputer-readable storage medium of claim 10, wherein the predicted AOVmovement comprises a predicted movement of a tool of the AOV.
 16. Thenon-transitory computer-readable storage medium of claim 10, whereininstructions for retraining the machine-learned model further cause theprocessor to: updating, by online learning techniques, themachine-learned model based on the actual movement corresponding to thefirst movement instruction.
 17. The non-transitory computer-readablestorage medium of claim 10, wherein the actual AOV movement is comparedto the predicted AOV movement responsive to the AOV switching between anautonomous operation mode and a manual operation mode.
 18. Thenon-transitory computer-readable storage medium of claim 10, wherein theactual AOV movement corresponding to the first movement instruction isdetermined by measuring a set of action conditions based on collectedsensor data.
 19. A model training system, comprising: a hardwareprocessor; a non-transitory computer-readable storage medium storingexecutable instructions that, when executed by the hardware processor,cause the model training system to perform steps comprising: training,by the training system, a machine-learned model based on a training setof data that describes movement instructions provided to one or moreautonomous off-road vehicles (AOVs) and resulting AOV movements, themachine-learned model configured to predict an AOV movement resultingfrom a movement instruction; determining, by the training system, thatan actual AOV movement for a first movement instruction is differentthan a predicted AOV movement by a threshold movement amount; andretraining, by the training system, the machine-learned model such thata difference between a subsequent actual AOV movement and the intendedAOV movement is less than the threshold movement amount.
 20. The methodof claim 19, wherein the machine-learned model is further configured togenerate a control signal for actuating a control surface of the AOVbased on the predicted AOV movement resulting from the movementinstruction.